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ABSTRACT 


This  mfimorandiim  reports  the  proced\ires 
used  for  the  development  and  scoring  of  hands- 
on  performance  tests  for  the  mechanical  mainte¬ 
nance  phase  of  the  Job  Performance  Meas\u*e- 
ment  ( JPM)  Project. 


EXECUTIVE  SUMMARY 


The  Job  Performance  Measurement  (JPM)  Project  is  a  multiyear  effort  to 
validate  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  against  hands-on 
measures  of  job  performance.  The  project  is  testing  representative  military 
occupational  specialties  (MOSs)  within  each  of  the  four  Marine  Corps  aptitude 
composites.  For  the  mechanical  maintenance  composite,  automotive  mechanics 
(MOS  3521)  and  four  helicopter  specialties  (CH-46,  MOS  6112;  CH-53A/D, 
MOS  6113;  UH/AH,  MOS  6114;  and  CH-53E,  MOS  6115)  were  tested.  This  research 
memorandnm  details  the  test  development  process  that  was  used  to  construct 
objective,  accurate,  and  representative  measures  of  job  performance  for  each  of  these 
MOSs.  The  test  development  process  had  four  stages:  specification  of  the  job 
requirements  domain,  sampling  representative  content,  constructing  objective 
hands-on  tests,  and  establishing  standardized  scoring  procedures. 

SPECIFYING  JOB  REQUIREMENTS 

Without  a  standardized  and  objective  procediure  for  defining  job  requirements, 
JPM  hands-on  tests  could  be  criticized  as  not  adequately  representing  what  indivi¬ 
duals  are  required  to  do  on  the  job.  The  usefulness  of  JPM  results  depends  on  the 
extent  to  which  the  hands-on  tests  accurately  reflect  the  tasks  and  behaviors  required 
by  mechanics  on  the  job.^  Therefore,  the  basis  for  defining  the  job  requirements  for 
each  MOS  was  the  Individual  Training  Standards  (ITSs)  for  the  automotive  me¬ 
chanics  and  the  ITSS  MATMEP  (Individual  Training  Standards  System:  Maintenance 
Training  Management  and  Evaluation  Program)  for  helicopter  mechanics.  These 
documents  serve  as  Marine  Corps  doctrine  on  specific  job  requirements,  at  the  task 
level,  for  each  of  the  mechanical  MOSs.  Extensive  job  analyses  were  conducted  based 
on  these  training  materials  to  organize  the  job  requirements  domain  and  thus  facil¬ 
itate  the  sampling  of  test  content.  Marine  Corps  subject  matter  experts  (SMEs) 
extensively  reviewed  and  modified  the  results  of  the  job  analysis.  At  the  end  of  the 
analysis,  the  job  domains  were  organized  into  matrices  that  specified  the  systems 
(e.g.,  electrical,  hydraulic)  and  mechanical  functions  (e.g.,  troubleshoot,  remove,  and 
replace)  required  of  mechanics  in  conducting  their  jobs. 


1.  CNA  Research  Contribution  570,  Developing  a  Competency  Scale  for  Hands-on  Measures 
of  Job  Proficiency,  by  Paul  W.  Mayberry,  Dec  1987. 
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SAMPLING  OF  JOB  REQUIREMENTS 


Each  MOS’s  job  reqtiirements  included  several  thousand  tasks — far  too  many  to 
be  performed  within  the  eight-hoiir  period  allotted  for  testing.  Therefore, 
Marine  Corps  SMEs  rank-ordered  the  importance  of  systems  and  mechanical 
functions  for  each  MOS  according  to  how  representative  and  difficult  they  were. 

The  representativeness  ratings  reflected  the  degree  to  which  performance  in 
one  system  or  function  would  imply  proficient  performance  in  other,  lower-ranked, 
systems  or  functions.  The  difficulty  ratings  were  judgments  on  how  difficult  it  is  for 
an  average  individual  to  be  trained  to  perform  proficiently  on  a  particular  system  or 
function  (i.e.,  time  to  learn).  Rating  exercises  ranked  systems  and  functions  separate¬ 
ly  on  these  two  dimensions  as  well  as  on  which  system  was  the  most  representative/ 
difficult  within  each  function  and  which  function  was  the  most  representative/difficult 
within  each  system.  The  outcome  of  these  rating  exercises  identified  “marker”  cells 
within  a  function-by-system  matrix  that  reflected  content  areas  most  representative 
of  p)erformance  in  the  entire  domain  of  job  requirements.  Difficulty  ratings  for  each 
cell  were  also  available  to  assist  in  sampling  of  test  content. 

Despite  emphasis  on  the  marker  cells  the  niimber  of  tasks  within  each  cell  was 
still  considerable.  Fiuther  analyses  were  conducted  to  identify  the  behaviors 
associated  with  performing  each  task  within  the  cell.  These  behavioral-element-by¬ 
task  matrices  readily  identified  the  degree  of  common  behaviors  across  tasks.  A 
samphng  plan  was  devised  to  select  tasks  for  each  MOS  so  that,  in  the  aggregate,  a 
maximiim  niimber  of  different  behaviors  were  tested.  Additional  information  (such 
as  time  to  perform,  equipment  and  spare  parts  needed,  and  special  considerations) 
was  obtained  for  the  sampled  tasks  to  assist  in  the  construction  of  hands-on  tests. 

CONSTRUCTING  HANDS-ON  TESTS 

Detailed  analyses  were  conducted  for  each  task  selected  for  hands-on  testing. 
These  analyses  supplemented  the  training  materials  to  identify  the  critical  steps 
required  to  perform  the  task.  Particular  attention  was  devoted  to  being  able  to 
objectively  score  the  step  as  successfully  p)erformed  or  not.  Ambiguous  or  non¬ 
observable  steps  were  either  revised  or  deleted.  Some  tasks  reqiiired  modification  so 
that  one  individual,  not  a  team  of  mechanics,  could  p)erform  the  tasks.  Certain  steps 
of  some  of  the  longer  tasks  were  eliminated  to  reduce  testing  time.  For  these  tasks, 
the  testing  situation  was  preset  to  begin  at  the  point  where  the  most  critical  per¬ 
formance  would  be  scored  (e.g.,  lug  nuts  and  wheel  were  already  off  for  “remove  and 
replace  the  brake  shoes”). 
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The  task  analysis  attempted  to  produce  initial  versions  of  the  hands-on  tests 
that  were  faithful  to  the  training  materials  on  which  they  were  based  and  that  could 
be  scored  objectively.  The  next  step  was  to  conduct  tryouts  of  each  hands-on  task 
with  job  incumbents  to  be  sure  that  Marine  examinees  would  understand  what  was 
expected  of  them  and  that  test  administrators  could  score  each  performance  step 
accurately.  After  performing  each  task,  the  job  incumbent  was  debriefed  to  deter¬ 
mine  if  he  understood  the  instructions  and  was  asked  to  evaluate  the  appropriate¬ 
ness  of  the  performance  steps  on  which  he  was  graded  relative  to  how  he  performed 
the  task  on  the  job.  Mvdtiple  iterations  of  this  process  were  conducted  with  several 
job  inciimbents  until  there  was  general  consensus  on  the  clarity  of  the  test,  its 
fidehty  to  actual  job  performance,  and  the  scorabihty  of  its  steps.  In  addition, 
judgments  were  made  concerning  how  to  modify  tasks  to  be  appropriate  for  indivi¬ 
dual  testing,  fit  within  reasonable  time  limits,  and  avoid  redundant  steps. 

SCORING  OF  HANDS-ON  TESTS 

Hands-on  assessment  is  individualized  testing — one  examinee  performing  for 
one  test  administrator.  The  abihty  of  the  test  administrator  to  score  performance 
consistently  across  all  mechanics,  over  time,  and  for  all  test  content  is  central  to  the 
proper  conduct  of  this  testing.  Recognizing  the  pivotal  role  of  the  test  adminis- 
trator,!  the  same  team  of  trainers  conducted  all  training,  using  extensive  training 
materials  that  were  prepared  to  ensure  standardization  of  training  both  within  and 
across  test  sites.  Each  test  administrator  completed  two  weeks  of  training.  The 
first  week  was  spent  learning  the  specific  content  of  the  hands-on  tests  and  how  to 
administer  tasks  in  a  standardized  manner.  Test  administrators  conducted  much 
role  playing  as  they  learned  to  observe,  score,  and  time  hands-on  performance.  The 
second  week  was  a  complete  trial  of  the  full  testing  evolution  using  the  actual 
mechanics.  Although  this  was  a  practice  round,  test  conditions  were  the  same  as 
they  would  be  for  the  full-scale  testing.  Test  administrator  scoring  was  closely 
monitored  to  identify  problems  immediately  and  to  address  questions.  At  the  end  of 
each  day  of  this  trial  testing,  test  administrators  discussed  difficulties  that  they  had 
and  unique  circumstances  that  had  not  been  addressed  in  the  training. 

The  measurement  of  mechanical  proficiency  is  complicated  by  the  fact  that  an 
experienced  mechanic  might  perform  certain  aspects  of  tasks  correctly  without 
following  the  step-by-step  procediues  of  the  hands-on  test,  which  were  developed 
from  the  technical  manuals.  The  mechanic  would  therefore  receive  low  test  scores. 


1.  Test  administrators  were  retired  and  former  Marines  with  relevant  mechanical  experi¬ 
ence  and  expertise  in  working  closely  with  yoimg  Marines. 
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An  important  consideration  for  this  project  was  the  distinction  between  the  process 
(percentage  of  steps  performed  correctly)  and  the  product  (whether  the  final  outcome 
was  correct).  Additionally,  mechanics  were  timed  in  the  performance  of  tasks  in 
order  to  measirre  their  efficiency. 

SUMMARY 

The  procedirres  used  for  the  development  and  scoring  of  the  mechanical  hands- 
on  performance  tests  ensirred  that  vahd  inferences  about  a  Marine’s  job  performance 
could  be  drawn  from  hands-on  test  scores.  ITSs  were  used  as  a  foundation  for 
defining  job  requirements;  Marine  Corps  job  experts  rated  each  job  requirement  for 
how  representative  it  was  to  the  job  as  a  whole.  A  random  samphng  procedirre 
selected  the  tasks  to  be  used  from  those  judged  to  be  most  representative. 

After  tasks  were  selected,  trial  testing  allowed  a  prehminary  assessment  of 
tasks’  feasibility,  time  limits,  and  procedirres.  Retired  and  former  Marine  mechan¬ 
ics  were  hired  and  trained  for  two  weeks  in  preparation  for  serving  as  test  adminis¬ 
trators.  Continual  monitoring  during  testing  minimized  the  amoimt  of  missing  data 
or  drift  in  scoring  criteria. 

The  scoring  system  ensirred  that  experienced  mechanics  would  not  be  penahzed 
for  using  procedures  that  were  more  efficient  than  those  specified  by  the  technical 
manuals.  Quality  control  checks  were  made,  and  missing  data  were  imputed  where 
warranted.  The  consistency  of  hands-on  measirrement  across  testing  sites  was  also 
examined.  No  diSerences  between  sites  were  foimd  for  the  automotive  mechanics. 
Site  differences  were  found  for  helicopter  mechanics  that  were  not  reflective  of 
actual  differences  in  proficiency.  These  diSerences  were  adjusted  in  accordance 
with  suggestions  by  Marine  Corps  manpower  experts. 

The  result  of  these  procedures  was  to  ensirre  that  scores  from  the  mechanical 
maintenance  phase  of  the  Job  Performance  Measirrement  (JPM)  project  are  highly 
reflective  of  performance  in  the  actual  job  domain. 
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INTRODUCTION 


The  Job  Performance  Measurement  (JPM)  Project  is  a  miiltiyear  effort  to 
validate  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  against  hands-on 
measimes  of  job  performance.  To  validate  the  ASVAB  adequately,  test  development 
and  scoring  must  meet  high  technical  standards  as  well  as  satisfy  the  scrutiny  of  the 
Marine  Corps  as  being  representative  measures  of  job  performance.  This  memo¬ 
randum  details  the  specification  of  job  requirements,  the  samphng  of  elements  from 
job  requirements,  test  development,  and  the  scoring  of  the  hands-on  tests,  which 
were  administered  in  the  mechanical  maintenance  phase  of  the  JPM  project,  in 
accordance  with  earher  CNA  work  [1]. 

The  mechanical  maintenance  phase  of  the  project  presented  uruque  challenges 
to  test  development  and  test  scoring.  Both  automotive  and  hehcopter  mechanics  are 
expected  to  perform  an  extremely  large  number  of  tasks — numbering  in  the 
thousands.  Since  only  eight  hours  of  testing  was  allowed,  just  a  small  fraction  of  the 
total  nimiber  of  tasks  coiild  be  tested.  Therefore,  the  procedures  for  specifying  the  job 
domain  and  for  sampling  firom  the  job  domain  were  particularly  important.  Fimther- 
more,  because  experienced  mechanics  can  often  skip  steps  prescribed  in  the  technical 
manuals  and  stiU  successfully  perform  a  particiilar  task,  scoring  posed  a  particular 
challenge  for  this  phase  of  research.  Therefore,  “product”  scores,  which  indicated 
whether  the  task  had  been  successfully  completed,  were  introduced;  in  addition,  task 
performance  was  timed  so  that  each  mechanic’s  efficiency  covild  be  measimed. 

The  figure  below  shows  the  stages  of  developing  and  scoring  the  mechanical 
hands-on  performance  tests,  and  the  products  produced  at  each  stage.  The  first 
stage  is  specifying  the  job  requirements  domain.  Without  a  standardized  and 
objective  procedure  for  defining  job  requirements  for  each  MOS,  hands-on  tests 
coiild  be  criticized  as  not  adequately  representing  what  individuals  are  required  to 
do  on  the  job.  It  was  determined  that  the  critical  components  of  a  mechanic’s  job 
coiild  be  represented  by  the  systems  they  work  on  (e.g.,  brakes,  engines)  and  the 
functions  they  perform  (e.g.,  troubleshoot,  remove  and  replace).  Therefore,  system- 
by-function  matrices  were  developed  for  each  MOS. 

In  stage  2,  test  content  must  be  randomly  sampled  from  the  matrices  developed  in 
stage  1.  To  accomphsh  this,  subject  matter  experts  (SMEs)  rated  how  representative 
and  difficult  cells  in  the  system-by-function  matrix  were.  lists  of  tasks  from  the  most 
representative  cells  were  developed,  and  behavioral  similarities  across  tasks  (e.g., 
selecting  proper  tools,  reading  instruments)  were  listed.  Test  content  was  randomly 
selected  to  maximize  the  niunber  of  behaviors  that  were  tested  within  the  allotted  time. 
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Figure  1.  Stages  in  the  development  of  JPM  mechanical  maintenance  tests 


In  stage  3,  hands-on  tests  were  created  from  the  selected  content.  To  do  this, 
instructions  were  written  that  described  each  task’s  setup,  eqtiipment,  safety,  and 
time  reqtiirements.  Instructions  to  examiaees,  score  sheets,  and  iustructions  for 
test  administration  were  also  developed. 

Stage  4,  prehminary  trial  testing,  was  a  “reality  check”  on  the  tests  created. 

Members  of  the  test-development  staff  administered  the  test  to  a  small  number  of 
Marine  mechanics.  This  process  allowed  determinations  to  be  made  concerning  the 
time  allowances,  parts  lists,  instructions,  and  scoring  procedures.  Revisions  were 
made  on  the  basis  of  this  preliminary  test. 

After  stage  4,  tasks  were  revised  and  test  administrators  were  trained  for  a 
week  with  the  new  materials.  In  stage  5,  the  newly  trained  test  administrators 
conducted  a  “dress  rehearsal”  of  the  final  testing  procedures.  This  review  included 
all  test  site  setups  and  a  command  review  by  the  units  providing  mechanics  for 
testing.  A  month-long  “stand-down”  period  was  then  allowed  so  that  final  revisions 
of  test  procedures  could  be  completed  before  the  actual  testing. 

For  all  of  these  stages,  the  general  design  was  similar  for  the  groxind  and 
helicopter  mechanic  MOSs.  However,  because  USMC  doctrine  differentiates  the 
two  sets  of  MOSs,  shghtly  different  approaches  were  necessary.  The  Marines 
specify  that  an  organizational  automotive  mechanic  (MOS  3521)  will  work  on  a  , 

variety  of  different  types  of  vehicles.  In  contrast,  each  helicopter  mechanic  MOS 
corresponds  to  a  specific  category  of  helicopter.  Because  of  this  distinction,  an  effort 
was  made  to  choose  helicopter  tasks  that  were  comparable  across  aircraft  systems.  j 
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DEVELOPMENT  OF  THE  AUTOMOTIVE  MECHANIC  TEST 


Specifying  the  Job  Requirements  Domain 

To  define  the  MOS  3521  performance  domain,  it  was  essential  that  a  matrix  be 
developed  that  would  define  the  systems  and  mechanical  functions  across  vehicles. 
The  Individual  Training  Standards  (ITS)  document  for  MOS  3521,  which  lists  tasks, 
grouped  within  duty  areas,  was  used  as  the  initial  basis  for  specifying  job  require¬ 
ments.  Technical  manuals,  parts  manuals,  and  programs  of  instruction  (POls)  were 
also  used  as  supplements  for  defining  the  job  requirements. 

The  ITS  document  lists  15  duty  areas,  with  2  to  20  tasks  in  each  area.  Never¬ 
theless,  it  proved  to  be  inadequate  for  a  full  specification  of  the  MOS  3521  job 
domain  for  three  reasons  [2].  First,  the  Marine  Corps  had  yet  to  officially  certify  the 
ITSs,  although  major  commands  had  aheady  reviewed  them  extensively.  Second, 
the  descriptions  differed  in  their  levels  of  specificity;  that  is,  some  descriptions  were 
rather  simple,  others  were  extremely  complex.  Last,  the  ITS  document  grouped 
some  tasks  on  the  basis  of  activities;  others  were  grouped  by  equipment. 

To  supplement  the  materials  of  the  ITS  document  for  developing  a  comprehensive 
specification  of  job  requirements.  Marine  Corps  subject  matter  experts,  technical 
manuals,  and  maintenance  allocation  charts  (MACs)  were  also  consulted.  This 
synthesis  of  sources  resulted  in  three  broad  categories  useful  in  evaluating  the  3521 
job  domain:  the  type  of  vehicles  they  work  on,  the  vehicle  systems  they  work  on,  and 
the  functions  that  mechanics  perform  on  the  vehicles.  Table  1  presents  a  Hst  from 
these  groupings. 

Panels  of  Marine  Corps  SMEs  reviewed  the  lists  of  vehicles,  systems,  and 
functions.  They  agreed  that  the  five  primary  vehicles — M1008,  M998,  M923,  M813, 
and  LVS— were  the  most  important  for  the  job  of  a  3521,  and  the  most  frequently 
maintained,  repaired,  and  serviced  by  the  typical  automotive  mechanic.  All  3521s 
have  some  responsibihty  for  these  five  vehicles.  In  contrast,  a  3521’s  experience 
with  additional  vehicles  (e.g.,  tactical  trailers  or  refuelers)  depends  on  the  mechan¬ 
ic’s  unit  assignment.  Therefore,  it  was  decided  that  testing  should  focus  on  the  five 
primary  vehicles. 
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Table  1.  Categories  for  mapping  the  MOS  3521  job  domain 


Vehicles 


Primary  Vehicles 

Additional  Vehicles 

M998, 1-1 /4-ton  4x4  vehicle 

Ml  008, 1  -1 /4-ton  4x4  cargo  truck 
M813A,  800  series  5-ton  cargo  truck 
M923,  900  series  5-ton  cargo  truck 
LVS,  Logistical  Vehicle  System 

Tactical  trailers 

Trailer  refuelers 

Engine  cleaners 

Lubricating  and  servicing  unit 
Wreckers 

Systems 

Brake  systems 

Drive  trains 

Steering  systems 

Fuel-delivery  systems 

Body  systems 

Hydraulic  systems 

Recovery/lifting  systems 

Electrical  systems 

Diesel  engine 
Rear-suspension  systems 
Front-suspension  systems 
Chassis  systems 

Veicle  accessories 

Functions 

Inspect 

Troubleshoot 

Sen/ice 

Align/adjust 

Test 

Remove/replace 

The  panels  reviewed  the  original  13  systems  and  the  list  of  six  mechanic  functions. 
They  rated  the  six  functions  in  terms  of  representativeness  and  time  spent  on  the 
function.  They  then  identified  omissions  and  exceptions  to  tasks  noted.  For  example, 
3521s  do  not  repair/replace  starters  on  the  M998  vehicle.  At  the  end  of  this  process,  the 
panels  agreed  that  the  job  performance  domain  could  be  accurately  described  in  terms 
of  vehicles,  vehicle  systems,  and  mechanic  functions.  They  felt  that  job  reqioirements 
concerning  administrative  functions — dealing  with  manuals  and  forms — should  also 
be  included  in  the  testing.  Finally,  they  agreed  that  a  system-by-mechanic-function 
matrix  (table  2)  could  be  developed  for  each  of  the  five  vehicles  [2]. 

Sampling  Test  Content 

The  80  cells  in  table  2  represent  the  entire  3521  job  domain.  However,  because 
multiple  tasks  could  be  developed  within  each  cell,  table  2  stiU  represents  much  more 
material  than  co\ild  be  tested  in  an  eight-hour  period.  Because  it  wo\ild  be  impossible 
to  test  the  himdreds  of  tasks  represented  by  the  cells,  it  was  imperative  to  sample 
firom  the  matrices  while  maintaining  representativeness  to  the  entire  job  domain. 
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Table  2.  Boundaries  of  the  MOS  3521  job  performance  domain  [2] 


Mechanic 

functions 

Systems 

Brake 

system 

Electric 

system 

Drive 

train 

Diesel 

engine 

Steering 

system 

Rear 

suspen¬ 

sion 

system 

Front 

suspen¬ 

sion 

system 

Body 

system 

Chassis 

system 

Hydraulic 

system 

LVS 

Vehicle 

acces¬ 

sories 

Towing/ 

lifting 

acces¬ 

sories 

Fuel 

delivery 

system 

Inspect 

Service 

Troubleshoot/ 

diagnose 

Test 

Adjust/align 

Repair  and 
replace 

NOTE:  Each  cell  in  the  matrix  contains  a  number  of  tasks  for  each  of  five  vehicles: 
M998,  M1008,  M813,  M923,  and  LVS. 


Administrative 

functions 


Forms  Manuals 


Super  Cells 


As  a  preliminary  step  of  sampling,  the  panels  ranked  the  13  systems  for  » 

representativeness  to  MOS  3521  with  respect  to  each  of  the  functions  and  ranked 
the  6  functions  with  respect  to  each  of  the  systems.  There  was  a  great  deal  of 
agreement  that  six  functions  and  six  systems  constituted  the  crucial  and  representa¬ 
tive  core  of  the  MOS  3521  mechanic  job.  The  panel  felt  that  if  a  mechanic  coiild 
perform  the  six  functions  on  the  six  systems,  they  could  work  on  any  of  the  systems. 

The  36  cells  in  the  6x6  matrix  were  called  “super  cells.” 

Marker  Cells 

The  panels  were  also  asked  to  respond  to  the  question  “If  you  were  to  test  a  3521 
on  jxist  one  of  these  systems  for  this  function,  which  one  would  you  select?”  They  were 
then  asked  the  same  information  about  functions:  “If  you  were  able  to  test  on  just  one 
function  for  this  system,  which  one  would  you  select?”  The  cells  that  SMEs  picked 
most  often  as  the  most  representative  were  denoted  as  “marker  cells”  [2].  Table  3 
shows  these  cells,  summarizing  the  relationship  of  the  total  3521  job  performance 
domain  to  the  smaller  testing  domain,  and  identifies  the  12  marker  cells  with  the 
entire  job  domain.  The  testing  domain  consisted  of  the  36  super  cells,  whereas  the 
12  marker  cells  represent  the  portion  of  the  testing  domain  to  be  given  special 
emphasis  because  of  the  particiilar  importance  that  SMEs  attached  to  those  cells. 

Upon  further  review  of  the  functions,  the  SME  panels  determined  that  the 
troubleshoot  function  was  partially  redundant  with  certain  aspects  of  the  inspect  and 
test  functions.  Therefore,  it  was  decided  for  testing  purposes  to  include  tasks  firom  the 
“inspect”  and  “test”  marker  cells  as  part  of  troubleshooting  tasks  wherever  possible. 

Developing  Behavioral-Element  Matrices 

Once  the  job  performance  domain  was  established,  it  was  important  to  examine 
systematically  the  tasks  associated  with  each  marker  cell.  The  behaviors  reqiiired 
in  performing  each  task  were  identified,  and  the  comparability  of  such  behaviors 
across  tasks  was  established.  In  this  manner,  the  behavioral  requirements  for 
mechanical  performance  were  identified,  irrespective  of  tasks.  A  behavioral-  f 

element-by-task  matrix  was  created  for  each  marker  cell.  Table  4  provides  an 
example  for  the  “test  electrical  system”  cell.  The  SME  panels  reviewed  and  revised 
these  matrices  and  deleted  tasks  or  behaviors  if  they  (1)  could  not  be  tested  '' 

(e.g.,  were  too  long  or  dangerous);  (2)  required  more  than  one  mechanic  to  perform; 
or  (3)  required  equipment  that  would  not  be  available  to  a  normal  motor  pool. 
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Table  3.  MOS  3521  job  performance  domain  and  testing  domain  [2] 


Systems 

Mechanic 

functions 

Brake 

system 

Electric 

system 

Drive 

train 

Diesel 

engine 

Steering 

system 

Rear 

suspen¬ 

sion 

system 

Front 

suspen¬ 

sion 

system 

Body 

system 

Chassis 

system 

Hydraulic 

system 

LVS 

Vehicle 

acces¬ 

sories 

Towing/ 

lifting 

acces¬ 

sories 

Fuel 

delivery 

system 

Inspect 

# 

Service 

Troubleshoot/ 

diagnose 

• 

# 

Test 

# 

Adjust/align 

Repair  and 
replace 

# 

# 

Performance  testing  domain 


Forms  Manuals 

Administrative 

functions 


Marker  cells 


Table  4.  Example  of  task-by-behavioral-element  matrix  [2] 


Test  voltage  with 
multimeter 

Test  electrolyte 
specific  gravity 

Test  electrolyte 
temperature 

Test  continuity 
with  multimeter 

Behavioral  element 

998 

1008 

813 

923 

LVS 

998 

1008 

813 

923 

LVS 

998 

1008 

813 

923 

LVS 

998 

1008 

813 

923 

LVS 

Follow  specific  safety  instructions 

Set  switch  to  proper  position 

Interpret  instrument  reading 

Calibrate  meter  scale  (if  required  by  meter) 
Connect  probes  across  items 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Observe  operation  of  electrical  component 
Install  meter  probes  on  meter 

Install/remove  jumper  wires 

Operate  vehicle  controls  and  observe  results 

Prepare  test  equipment 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Follow  general  safety  precautions 

Select  proper  tools 

Fill  out  forms 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Read  manual 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Locate  information  in  manual 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

i 


4 


Task  Selection 


Tasks  were  weighted  by  the  nvunber  of  behavioral  elements  they  represented. 
Based  on  these  weightings,  tasks  within  a  marker  cell  were  randomly  chosen,  so 
that  tasks  with  a  larger  number  of  behavioral  elements  had  a  greater  probability  of 
selection.  When  a  task  was  randomly  selected,  the  remaining  tasks  were  re¬ 
weighted  to  exclude  previously  sampled  behavioral  elements  in  an  attempt  to 
maximize  the  number  of  different  behaviors  to  be  tested.  Because  the  matrices  were 
“dense,”  the  behavioral  elements  were  usually  exhausted  before  all  tasks  in  a  cell 
were  selected.  The  subject  matter  experts  reviewed  the  resulting  list  of  tasks  for 
representativeness  and  feasibility  of  testing. 

Creating  Testable  Units 

Once  tasks  had  been  randomly  selected,  detailed  descriptions  of  the  tasks  and 
the  performance  steps  required  to  accomplish  them  were  developed.  These  analyses 
included  the  materials  required  for  each  task,  the  sequence  of  steps,  and  any 
special  testing  or  safety  conditions.  Important  considerations  at  this  point  were  to 
ensure  that  appropriate  materials  needed  to  perform  a  task  would  be  available  and 
that  tasks  could  be  performed  within  the  time  allotted.  Therefore,  parts  lists  and 
tool  requirements  were  an  important  part  of  the  effort. 

Some  tasks  required  modification  so  that  one  individual,  not  a  team  of  mechan¬ 
ics,  could  perform  the  tasks.  Certain  steps  of  some  of  the  longer  tasks  were  ehmi- 
nated  to  reduce  testing  time.  For  these  tasks,  the  testing  situation  was  preset  to 
begin  at  the  point  where  the  most  critical  performance  would  be  scored  (e.g.,  lug 
nuts  and  wheel  were  already  off  for  “remove  and  replace  the  brake  shoes”).  Another 
consideration  was  the  development  of  a  preliminary  scoring  format  that  allowed  test 
administrators  to  determine  whether  steps  had  been  performed  correctly. 

Preliminary  Trial  Testing 

The  previous  activities  defined  a  set  of  tasks  and  testing  procedures,  but  these 
tests  had  not  been  administered  to  actual  Marines.  Therefore,  preliminary  trial 
tests  were  conducted  to  serve  as  a  “shakedown”  and  as  a  verification  of  required 
equipment,  station  setup,  and  scoring  procedures. 

The  preliminary  trial  testing  allowed  a  number  of  problems  to  be  identified  and 
corrected,  and  established  the  estimated  time  to  complete  tasks,  parts  and  tools 
lists,  and  manual  extracts.  It  allowed  administrators  to  determine  whether  all 
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selected  tasks  could  be  finished  within  the  specified  time  limits,  or  whether  further 
tasks  woiild  need  to  be  dropped. 

Incomplete  or  ambiguous  instructions  were  revealed  by  the  questions  of  the 
examinees.  Deficiencies  in  station  setup  and  equipment  hsts  became  apparent.  The 
trial  testing  also  assisted  with  the  “packaging^  of  tests  into  test  stations  and  helped 
identify  ambiguities  in  the  scoring  procedures,  performance  steps  that  were  difficvJt 
or  impossible  to  observe,  steps  that  covJd  be  effectively  combined,  and  the  best 
positions  for  administrators  to  take  when  scoring. 

Final  Test  Tryouts 

Once  the  procedures  and  scoring  protocols  were  revised  based  on  the  pre¬ 
liminary  trial  testing,  test  administrators  were  trained  in  the  scoring  procedures. 
At  the  end  of  the  training,  final  test  tryouts  were  conducted  for  one  week  at  both 
west  coast  and  east  coast  sites  to  prepare  the  administrators  for  the  actual  testing. 
These  tryouts  were  a  final  opportiinity  to  review  the  clarity  of  instructions,  the 
assignments  of  tests  to  stations,  the  timing  of  individual  tests,  and  the  feasibihty  of 
administering  tests  within  the  allotted  time.  The  effectiveness  of  administrator 
training  covJd  be  determined  as  well. 

A  command  review  was  performed  during  the  final  test  tryout,  so  that  com¬ 
manding  officers  covJd  see  the  procedures  and  make  recommendations.  After  the 
final  tryout,  the  tests  were  further  revised  during  a  stand-down  period  of  one  month 
before  actual  testing  began. 

Content  Not  Tested 

The  final  test  tryouts  provided  a  last  opportunity  to  modify  or  delete  tasks  that 
covJd  not  be  fitted  into  the  allotted  testing  time.  Overall,  few  behavioral  elements 
were  not  tested.  Appendix  A  (Tables  A-1  through  A-6)  shows  the  behavioral  ele¬ 
ments  tested  for  each  marker  cell.  As  explained  earlier,  subject  matter  experts  had 
identified  the  marker  cells  as  the  content  most  representative  of  performance  in  the 
respective  systems  and  functions.  Appendix  A  shows  that  essentially  aU  behavioral 
elements  in  the  marker  cells  were  tested  despite  testing  relatively  few  tasks.  It  also 
shows  that  when  a  behavioral  element  was  not  tested  within  one  marker  cell 
(e.g.,  “look  for  wear^  in  the  inspect  brakes  cell),  it  was  often  covered  by  a  task  firom 
another  cell  (e.g.,  troubleshooting  the  electrical  system).  Arrows  in  the  appendix 
tables  indicate  behavioral  elements  that  were  not  tested  by  any  tasks. 
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DEVELOPMENT  OF  HELICOPTER  MECHANIC  TESTS 


The  general  design  for  test  development  and  scoring  was  similar  for  the  groimd 
and  helicopter  mechanic  MOSs.  However,  because  Marine  Corps  doctrine  differen¬ 
tiates  the  two  sets  of  MOSs,  shghtly  different  approaches  were  necessary.  The 
Marines  specify  that  an  organizational  automotive  mechanic  (MOS  3521)  will  work 
on  a  variety  of  different  types  of  vehicles.  In  contrast,  helicopter  mechanics  have 
four  different  MOSs  corresponding  to  specific  categories  of  helicopter,  as  follows: 

•  MOS  6112 — CH/46  hehcopter 

•  MOS  6113— CH-53A/D  helicopter 

•  MOS  6114— AH-IJ,  AH- IT,  AH-IW,  and  UH-IN  helicopters 

•  MOS  6115 — CH-53E  hehcopter. 

As  a  result  of  these  differences,  it  was  important  that  the  hands-on  tests  for  hehcopter 
mechanic  include  tasks  that  reflected  the  unique  job  requirements  across  MOSs  while 
trying  to  maintain  comparable  test  content  to  assist  in  later  cross-MOS  performance 
comparisons. 

Specifying  the  Job  Requirements  Domain 

The  primary  source  used  for  specifying  the  job  domain  for  the  four  hehcopter 
mechanic  MOSs  was  the  ITSS  MATMEP  (Individual  Training  Standards  System: 
Maintenance  Training  Management  and  Evaluation  Program).  The  ITSS  MATMEP 
is  a  standardized  technical  skills  training  management  and  evaluation  program — it 
is  the  program  that  the  Marines  use  both  to  specify  and  to  evaluate  the  technical 
skiUs  required  of  hehcopter  mechanics. 

The  first  activity  was  to  determine  which  systems  were  common  across  the 
various  aircraft.  Sorting  the  MATMEPs  by  system  resulted  in  a  matrix  hke  that 
shown  in  table  5.  The  table  shows  that  a  number  of  systems  could  be  considered 
common  across  aircraft.  For  example,  ah  aircraft  appear  to  have  tasks  deahng  with 
flight  controls  and  powerplants.  Other  categories,  such  as  the  power  train,  had 
either  systems  or  subsystems  that  would  be  considered  comparable  across  aircraft. 
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Table  5.  Boundaries  of  the  MOS  61 1 2,  61 1 3,  61 1 4,  and  61 1 5  job  performance  domain  [2] 


Main  rotor 

Tail  rotor  system 

Helicopter 

Flight 

control 

system 

Main 

rotor 

system 

Rotor 

system 

Power 

train 

system 

Power 

plant 

system 

Tail 

rotor 

system 

Rotary 

rudder 

system 

Oil 

cooler 

system 

Fuel 

system 

Utility 

system 

Auxiliary 

power 

plant 

system 

Auxiliary 

fuel 

system 

External 

cargo 

hook 

system 

(1)  CH-46 

X 

X 

Sub 

X 

Sub 

X 

X 

X 

Sub 

Sub 

(2)  CH-53/A/D 

X 

X 

Sub 

X 

X 

Sub 

X 

X 

X 

Sub 

X 

(3)  CH-53E 

X 

X 

Sub 

X 

X 

Sub 

X 

X 

X 

Sub 

X 

(4)  UH-1N 

X 

X 

X 

X 

X 

X 

X 

X 

Sub 

Sub 

(5)  AH-1J 

X 

X 

X 

X 

X 

X 

X 

(6)  AH-1W 

X 

X 

X 

X 

X 

X 

X 

X 

(7)  AH-1T 

X 

X 

X 

X 

X 

X 

X 

X 

Key:  X  =  System  identified  in  MATMEP. 

Sub  =  System  is  a  subtask  of  a  larger  system  or  is  not  included  in  the  MATMEP. 


The  second  activity  was  to  determine  which  functions  mechanics  perform.  The 
duty  areas  in  the  MATMEPs  identified  12  common  mechanic  functions  across  the 
various  aircraft  systems,  as  shown  in  table  6.  This  table  shows  that  the  two  activi¬ 
ties  most  associated  with  troubleshooting — function  check  and  fault  isolation — are 
performed  on  all  helicopter  systems.  Remove/replace  is  another  function  performed 
on  all  helicopter  systems,  while  rig  and  adjust  are  performed  on  most  hehcopter  sys¬ 
tems.  All  other  functions  (e.g.,  scope,  set,  vibration  analysis)  apply  to  a  small  subset 
of  helicopter  systems. 

Next,  entries  within  the  matrices  needed  to  be  rated  so  that  their  relative 
representativeness  could  be  judged.  Subject  matter  experts  were  asked  to  rate  the 
aircraft  systems  according  to  the  amount  of  knowledge,  skills,  and  abilities  required 
to  perform  them.  The  experts  decided  that  troubleshooting  and  remove/replace 
functions  demanded  the  most  knowledge,  skills,  and  abilities.  This  decision,  which 
was  confirmed  by  a  second  expert  panel,  thus  identified  eight  “super  cells”  of  the 
most  representative  systems  and  mechanical  functions. 

Since  rig  and  adjust  were  considered  the  third  and  fourth  most  general  mechan¬ 
ical  functions,  cells  represented  by  those  functions  were  considered  “secondary  super 
cells.”  Table  7  shows  the  results  fixjm  the  various  panel  meetings.  This  matrix 
formed  the  basis  for  selecting  tasks  from  the  job  performance  domain  across  the  four 
MOSs. 

Sampling  Test  Content 

To  begin  selecting  test  content,  a  fist  of  tasks  was  developed  for  each  of  the 
super  cells  and  the  additional  secondary  super  cells  shown  in  table  7.  Then  the 
helicopter  technical  manuals  were  analyzed  to  identify  the  behavioral  elements  for 
each  task.  These  behavioral  elements  were  described  at  a  slightly  higher  level  of 
generality  than  the  step  level.  An  example  of  one  of  the  behavioral-element-by-task 
matrices  is  given  in  appendix  B. 

Tasks  were  then  randomly  selected  fixjm  the  super  cells  and  secondary  super 
cells  to  maximize  coverage  of  the  behavioral  elements  within  each  targeted  cell. 
This  procedure  produced  a  list  of  initial  tasks  to  be  considered  for  testing.  From  this 
list,  comparable  tasks  across  MOSs  were  identified,  as  shown  in  table  8. 
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Table  6.  A  function-by-system  matrix  for  helicopter  mechanics  [2] 


Rotor  systems 

Tail  rotor  system 

Helicopters: 

1-7 

2-7 

1 

1-7 

1-7 

4-7 

2.3 

1-7 

1-7 

1-4 

1.2,3 

1-7 

1.2,  3.  4 

Flight 

control 

system 

Main 

rotor 

system 

Rotor 

system 

Power 

train 

system 

Power 

plant 

system 

Tail 

rotor 

system 

Rotary 

rudder 

system 

Oil 

cooler 

system 

Fuel 

system 

Utility 

system 

Auxiliary 

power 

plant 

system 

Auxiliary 

fuel 

system 

External 

cargo 

hook 

system 

1.  Function  check 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

2.  Fault  isolation 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

3.  Remove/replace 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

4.  Rig 

X 

X 

X 

X 

X 

X 

X 

5.  Adjust 

X 

X 

X 

X 

X 

X 

X 

(4) 

X 

6.  Scope 

X 

7.  Set 

X 

v 

v 

Y 

Y 

8.  Tracks/balances 

A 

A 

A 

A 

Helicopters: 

9.  Static  balances 

X 

1  « 
2  » 
3  » 

CH-46 

CH-53A/D 

CH-53E 

5  «  AH-IJ 

6  «  AH-1W 

7  -  AH-IT 

Y 

Y 

Y 

Y 

4  « 

UH-1N 

10.  Shims 

A 

(2-5) 

A 

(2-7) 

11.  Repacks 

X 

12.  Vibration  analysis 

X 

X 

X 

X 

(4-7) 

X 

X 

X 

(6) 

4 


Table?.  MOS  6112,  6113,  6114,  and  61 15  job  performance  and  testing  domain  [2] 


Main  rotor 

Tail  rotor  system 

Helicopters: 

1-7 

2-7 

1 

1-7 

1-7 

4-7 

2.3 

1-7 

1-7 

1-4 

1.2,3 

1-7 

1.2,  3,4 

Flight 

control 

system 

Main 

rotor 

system 

Rotor 

system 

Power 

train 

system 

Power 

plant 

system 

Tail 

rotor 

system 

Rotary 

rudder 

system 

ai 

cooler 

system 

Fuel 

system 

Utility 

system 

Auxiliary 

power 

plant 

system 

Auxiliary 

fuel 

system 

External 

cargo 

hook 

system 

1.  Troubleshoot 

.X  , 

=  x' 

K 

X  - / 

•  X  ■ 

X 

X 

X 

X 

X 

X 

X 

X 

2.  Remove/replace 

■  ^ 

K 

i  X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

3.  Rig 

X  . 

X 

X 

X 

X 

X 

X 

4.  Adjust 

x- 

< 

X 

X 

. X . : 

X . 

X 

X 

(4) 

X 

5.  Scope 

k 

X 

6.  Set 

X 

Helicopters: 

1  =  CH-46  5  -  AH-U 

2  =  CH-53A/D  6  «  AH-1W 

3  «  CH-53E  7  -  AH-IT 

7.  Tracks/balanoes 

X 

X 

X 

X 

8.  Static  balances 

X 

4  = 

UM-IN 

=  Super  cell  1  Testing 

=  Secondary  super  cell  J 

— 

9.  Shims 

X 

X 

(2-5) 

X 

X 

(2-7) 

10.  Repacks 

X 

1 1 .  Vibration  analysis 

X 

X 

X 

X 

(4-7) 

X 

X 

X 

(6) 
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Table  8.  Comparable  tasks  tested  across  MOS  [2] 


6112 

CH-46 

6113 

CH-53A/D 

6114 

UH-1N/AH-1W 

6115 

CH-53 

TS  flight  controls 

TS  flight  controls 

TS  flight  controls 

TS  flight  controls 

R/R  unitary  pedal 

R/R  collective  pitch  bellcrank 
Auxiliary  power  plant 

R/R  mechanical  screwjack 

R/R  collective  pitch  bellcrank 
Auxiliary  power  plant 

R/R  unitary  pedal 

R/R  collective  pitch  bellcrank 

*  *  * 

R/R  mechanical  screwjack 
R/R  collective  pitch  bellcrank 
Auxiliary  power  plant 

TS  main  rotor 

TS  main  rotor 

TS  main  rotor 

TS  main  rotor 

R/R  rotor  head 

R/R  rotor  head 

R/R  rotor  head 

R/R  rotor  head 

R/R  rotating  scissors 

R/R  rotating  scissors 

R/R  drive  link 

R/R  rotating  scissors 

TS  PP  fuel  system 

TS  PP  oil  system 

TS  PP  control  system 

*  *  * 

TS  PP  oil  system 

*  *  * 

TS  PP  fuel  system 

TS  PP  oil  system 

TS  PP  control  system 

TS  fuel  system 

TS  oil  system 

*  *  * 

R/R  fuel  boost  pump 

R/R  fuel  boost  pump 

R/R  fuel  boost  pump 

R/R  fuel  boost  pump 

TS  MGB  oil  system 

TS  MGB  chiplights 

TS  MGB  drive  shaft 

*  *  * 

TS  MGB  chiplights 

TS  MGB  drive  shaft 

*  *  * 

TS  MGB  chip  detector 

*  *  * 

*  *  * 

TS  MGB  chiplights 

*  *  * 

R/R  PT  drive  shaft 

R/R  sump  oil  filter 

R/R  PT  drive  shaft 

R/R  sump  oil  filter 

R/R  PT  drive  shaft 

R/R  sump  oil  filter 

R/R  PT  drive  shaft 

R/R  sump  oil  filter 

Adjust  pitch  control  rod 

Adjust  flight  control  rod 

Adjust  pitch  control  rod 

Adjust  flight  control  rod 

*  *  * 

Adjust  flight  control  rod 

Adjust  pitch  control  rod 

Adjust  flight  control  rod 

Ground  handling 

Ground  handling 

Ground  handling 

Ground  handling 

IC 


Creating  Testable  Units 


^  The  selected  tasks  were  then  examined  and  described,  for  purposes  of  develop¬ 

ing  testable  units.  Testable  units  included  instructions  for  the  test  administrator, 
procedures  for  task  setup,  a  bst  of  materials  needed,  and  procedures  for  scoring  and 
,  administering  the  test.  The  scoring  procedure  listed  the  steps  examinees  were 

reqviired  to  perform  and  a  set  of  criteria  forjudging  examinee  performance. 

Preliminary  Trial  Testing 

The  previous  activities  defined  a  set  of  tasks  and  testing  procedures,  but  these 
tests  had  not  been  administered  to  actual  examinees.  Therefore,  prehminary  trial 
tests — the  same  as  those  used  for  the  automotive  phase,  described  earlier — were 
conducted  to  serve  as  a  “shakedown.” 

Final  Test  Tryouts 

Once  the  procedures  and  scoring  protocols  were  in  a  nearly  final  form, 
test  administrators  were  trained  in  the  scoring  procedures.  At  the  end  of  the  train¬ 
ing,  week-long  final  test  tryouts  were  conducted  to  prepare  the  administrators  for 
the  actual  testing.  Like  the  automotive  phase  of  JPM  testing,  the  final  test  tryouts 
and  command  review  for  the  hehcopter  mechanics’  testing  were  a  final  opportunity 
to  review  the  clarity  of  instructions,  the  assignment  of  tests  to  stations,  and  the 
timing  of  individual  tests. 

Content  Not  Tested 

The  final  test  tryouts  provided  a  last  opportunity  to  drop  tasks  that  could  not  be 
fitted  into  the  allotted  testing  time.  Once  the  last  deletions  were  made,  the  coverage 
of  the  remaining  tasks  was  assessed.  Overall,  few  behavioral  elements  were  not 
tested  in  at  least  one  task.  Appendix  C  fists  the  behavioral  elements  covered,  by 
MOS. 

SCORING  AND  QUALITY  CHECKS  FOR  MECHANICAL  HANDS-ON 
»  TESTS 

So  far,  this  memorandxim  has  described  the  process  of  developing  the  tests  for 
♦  mechanical  maintenance  specialties.  Given  all  the  attention  to  careful  test  develop¬ 

ment,  it  was  necessary  to  give  equally  careful  attention  to  scoring  and  quality  checks 
for  the  hands-on  tests. 
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Scoring  of  the  Hands-on  Tests 


Step  Scores 

Scoring  the  automotive  and  helicopter  hands-on  mechanical  tests  involved 
several  special  considerations.  The  first  was  that  an  experienced  mechanic  could 
skip  steps  specified  in  the  technical  manual  based  on  a  deeper  understanding  of  the 
procedures  involved  and  his  past  experiences.  For  example,  whereas  the  manual 
might  specify  loosening  several  screws  in  order  to  take  out  several  individual  parts, 
the  experienced  mechanic  may  know  that  loosening  one  main  screw  would  dis¬ 
connect  the  entire  component  and  save  time.  The  experienced  mechanic  might  also 
add  steps  that  are  not  specified  in  the  technical  manual.  For  example,  the  mechanic 
might  loosen  a  bolt  in  an  earlier  portion  of  the  task,  in  anticipation  of  a  later  step 
where  loosening  that  bolt  woxild  have  been  more  difBcxilt.  In  cases  such  as  these,  a 
strict  “go/no-go”  scoring  woxild  place  the  experienced  mechanic  at  a  disadvantage — 
the  mechanic  woxild  be  penalized  for  having  improved  upon  procedures  prescribed  in 
the  technical  manual. 

A  second  challenge  was  how  to  score  troubleshooting  tasks.  The  technical 
manual  specifies  a  series  of  general  steps  that  will  pinpoint  any  problem,  including 
those  that  occur  infi'equently.  However,  an  experienced  mechanic  knows  that 
certain  problems  are  more  likely  to  occur  than  others.  Rather  than  follow  general 
procedures,  an  experienced  mechanic  might  immediately  search  for  the  most 
common  problem,  ignoring  or  skipping  the  general  procedures  specified  in  the 
technical  manual.  If  the  troubleshooting  task  were  scored  strictly  “go/no-go” 
according  to  whether  technical  manual  steps  were  performed,  the  experienced 
mechanic  might  be  penalized  even  though  that  mechanic  correctly  diagnosed  the 
problem  in  less  time  than  it  woxild  take  the  less  experienced  mechanic. 

These  considerations  led  to  a  departxire  firom  the  traditional  “go/no-go”  format 
for  scoring  each  step.  Instead,  steps  were  scored  as  follows: 

•  “Go,”  if  the  step  was  performed  correctly 

•  “No-go,”  if  the  step  was  performed  incorrectly 


“Did  not  do,”  if  the  step  was  simply  skipped  or  ignored 


•  “Could  not  do,”  if  the  step  coidd  not  be  attempted  for  reasons  unrelated  to 
the  examinee’s  proficiency  (e.g.,  equipment  essential  to  step  completion  was 

V  broken  or  missing). 

In  addition,  each  task  was  timed  to  determine  how  quickly  it  was  accomphshed.^ 

,  Fxirthermore,  a  score  was  added  to  the  scoring  sheet  to  indicate  whether  the  task 

had  been  completed  correctly.  These  “product”  scores  could  be  used  to  give  credit  to 
the  person  who  skipped  many  steps  but  who  nevertheless  completed  the  assignment 
correctly.  A  final  innovation  was  the  inclusion  of  “not  in  the  manual”  steps,  which 
were  beneficial  to  perform  but  not  required  by  the  technical  manual.  For  example, 
in  the  “adjust/align  power  steering  assist  cylinder”  task,  the  manual  fails  to  mention 
that  loosening  a  clamp  screw  will  make  it  easier  to  set  the  distance  between  two 
studs.  Examinees  who  remembered  to  loosen  the  clamp  screw  were  rated  “go”  on 
the  not-in-the  manual  step  “loosened  clamp  screw.” 

Task-Level  Scores 

The  computation  of  task-level  scores  was  altered  to  take  advantage  of  the 
additional  flexibility  allowed  by  the  nontraditional  step  scoring.  To  compute 
task-level  scores,  product  steps  and  process  steps  had  to  be  distinguished. 
In  addition,  it  was  necessary  to  reconfirm  on  which  steps  “did  not  do”  should  be 
coimted  as  wrong,  and  on  steps  “did  not  do”  should  be  considered  neutral,  because 
not  doing  the  steps  did  not  adversely  affect  performance.  Based  on  feedback  from 
subject  matter  experts,  it  was  decided  that  for  most  steps,  “did  not  do”  should  be 
scored  as  wrong.  But  for  some  steps,  particularly  in  troubleshooting  tasks,  failure  to 
perform  should  be  considered  neutral. 

Such  a  scoring  system  allowed  greater  flexibility  in  interpreting  step  scores  for 
later  analysis.  For  example,  the  following  types  of  performance  indicators  could  be 
computed  using  this  scoring  system: 

•  Percentage  correct  on  essential  steps  (i.e.,  in  which  failure  to  perform  the 
step  is  counted  as  wrong) 

»  •  Percentage  correct  on  all  steps 

•  Percentage  of  tasks  with  correct  “product” 


1.  Marines  were  aware  that  they  were  being  timed,  but  they  were  given  no  instructions  or 
incentive  to  work  quickly. 
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•  Percentage  of  “extra”  (not-in-manual)  steps  correctly  performed 

•  “EfBciency”  defined  as  the  percentage  correct  on  essential  steps  divided  by 
time  to  complete  the  task. 

Quality  Checks 

The  hands-on  performance  tests  were  administered  to  about  1,000  automotive 
mechanics;  about  200  helicopter  mechanics  were  tested  for  each  of  the  respective 
aircraft.  Separate  grovmd  and  air  teams  of  test  administrators  conducted  the  testing 
at  the  east  and  west  coast  test  sites.  The  hands-on  performance  data  were  entered 
into  computers  at  each  test  site  to  identify  immediately  any  missing  data  and  to  allow 
for  monitoring  of  test  administrator  scoring  trends.  Once  all  on-site  data  checks  and 
quality  analyses  had  been  conducted,  the  data  were  transmitted  to  CNA  for  merging 
into  the  master  JPM  database  and  further  processing.  CNA  conducted  additional 
quality  control  tests  and  imputed  missing  data  for  individual  cases  as  was  warranted. 
These  data  quality  analyses  are  documented  elsewhere  [3,  4]. 

Initial  analyses  of  the  hands-on  data  focused  on  the  quality  of  measurement  by 
each  hands-on  task  by  examining  step-level  performance  data.  The  step  analyses 
(analogous  to  item  analyses  for  paper-and-pendl  tests)  examined  step-total  score 
correlations  and  step-task,  score  correlations  to  identify  steps  that  were  potentially 
ambiguous  or  provided  poor  measurement.  Low  values  on  either  of  these  correlation 
measures  indicated  that  the  step  was  not  consistently  measuring  the  same  concept 
as  the  overall  task  or  total  hands-on  scores.  For  the  automotive  mechanic  hands-on 
test,  8  steps  were  deleted  (out  of  441  steps  on  the  total  test),  and  for  the  helicopter 
hands-on  tests,  10  to  22  steps  were  deleted  (out  of  253  to  372  steps  on  the  total  test). 
Task  scores  were  then  computed  based  on  all  appropriate  steps,  and  the  scores  were 
averaged  to  produce  total  scores  for  each  MOS. 

Site  Comparisons 

A  second  concern  was  to  determine  the  consistency  of  hands-on  measurement 
across  testing  sites.  AU  MOSs  were  tested  on  both  the  east  and  west  coasts  except 
for  the  CH-53  A/D  mechanics  (6113),  who  were  tested  only  on  the  west  coast.  The 
relationship  between  tasks,  total  hands-on  score,  time  in  service  (TIS),  and  aptitude 
were  examined  separately  by  site  for  each  MOS.  The  correlations  among  these 
variables  were  fovmd  to  be  very  consistent  for  the  two  sites  for  all  MOSs.  However, 
large  differences  in  the  percent-correct  scores  for  the  total  test  were  noted  between 
the  two  sites  for  all  helicopter  MOSs  (except  MOS  6113,  which  was  tested  at  only 
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one  site)  (see  tables  9  through  11).  Similar  site  differences  did  not  occur  for  the 
automotive  mechanics. 


Table  9.  Hands-on  task  performance  scores  and  site  adjustments 
for  CH-46  mechanics  (MOS  6112) 


Task 

Site  A 

Site  B 

Site 

adjustment 

Mean 

STD 

Mean 

STD 

1A 

88.89 

10.06 

82.14 

15.18 

-7.77 

IB 

69.80 

17.17 

79.06 

27.10 

9.54 

2A 

95.54 

11.37 

78.62 

22.66 

-17.66 

2B 

90.45 

13.52 

60.96 

22.07 

-30.61 

2C 

93.78 

9.95 

78.73 

13.92 

-16.02 

2D 

79.42 

16.69 

77.03 

16.57 

-2.72 

3A 

91.03 

12.00 

69.86 

19.89 

-22.99 

3B 

97.11 

4.58 

87.81 

16.49 

-9.89 

3C 

96.63 

7.50 

85.06 

14.09 

-12.52 

4A 

95.06 

6.74 

84.99 

11.63 

-10.42 

4B 

74.95 

23.23 

56.73 

24.73 

-18.34 

5A 

93.07 

7.45 

90.57 

8.82 

-2.85 

5B 

82.01 

15.08 

81.89 

13.10 

-0.65 

6A 

86.52 

18.29 

87.61 

20.05 

-1 .05 

7A 

91.13 

13.40 

78.04 

17.10 

-13.56 

7B 

85.89 

22.38 

83.04 

18.97 

-4.59 

8A 

91.06 

12.32 

87.49 

12.84 

-4.52 

SB 

87.72 

12.78 

96.61 

9.90 

9.04 

8C 

84.13 

10.52 

86.94 

12.91 

2.52 

Forms 

60.78 

24.65 

71.15 

21.06 

8.61 

Manuals 

83.09 

16.37 

84.14 

12.05 

-0.31 

Total 

86.52 

6.61 

80.38 

8.00 

MM 

113.13 

7.70 

1 1 6.54 

10.20 

TIS 

50.79 

37.05 

53.01 

40.01 

N 

95 

79 

NOTE:  Site  adjustments  retain  performance  differences  due  to  aptitude  and  time 

in  service;  adjustments  are  added  to  site  A  hands-on  scores. 

% 
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Table  10.  HandS'HDn  task  performance  scores  and  site  adjustments 
for  U/AH-1  mechanics  (MOS  6114) 


Task 

Site  A 

She  B 

She 

adjustment 

Mean 

STD 

Mean 

STD 

1A 

91.41 

8.67 

88.12 

13.61 

-1.81 

1B 

72.92 

15.55 

63.05 

12.06 

-9.40 

2A 

94.92 

11.50 

93.71 

14.16 

-0.67 

3A 

97.29 

5.42 

51.11 

28.86 

^.94 

3B 

97.94 

6.52 

78.62 

20.49 

-17.89 

4A 

87.43 

11.85 

71.61 

22.17 

14.69 

5A 

89.16 

8.28 

74.46 

21.12 

-13.13 

5B 

78.10 

21.87 

36.90 

23.49 

-39.22 

6A 

90.48 

9.28 

85.27 

9.55 

-5.33 

6B 

84.52 

21.26 

79.11 

20.64 

-5.27 

7A 

74.78 

20.28 

62.43 

18.95 

-9.70 

7B 

93.71 

8.32 

87.32 

12.94 

-5.53 

8A 

86.67 

19.44 

72.56 

20.59 

-13.16 

8B 

85.65 

14.56 

70.73 

21.84 

-12.04 

8C 

76.48 

27.34 

67.13 

25.66 

-8.17 

Forms 

59.38 

20.40 

63.49 

20.55 

6.57 

Manuals 

83.17 

17.78 

85.91 

15.80 

5.06 

Total 

84.92 

6.69 

72.48 

8.01 

MM 

117.34 

9.08 

115.77 

8.51 

TIS 

59.25 

42.00 

39.69 

24.25 

N 

63 

151 

NOTE:  Site  adjustments  retain  performance  differences  due  to  aptitude  and  time 

in  service;  adjustments  are  added  to  site  A  hands-on  scores. 

T 


V 
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Table  11.  Hands-on  task  performance  scores  and  site  adjustments 
for  CH-53  E  mechanics  (MOS  61 15) 


Task 

Site  A 

Site  B 

Site 

adjustment 

Mean 

STD 

Mean 

STD 

1A 

92.10 

10.30 

73.88 

17.30 

-18.58 

1B 

79.03 

19.25 

59.87 

29.44 

-20.61 

1C 

97.33 

8.61 

87.64 

17.19 

-9.96 

1D 

76.70 

16.22 

77.26 

15.15 

0.85 

2A 

98.33 

3.76 

89.10 

12.40 

-7.94 

3A 

98.60 

3.88 

86.54 

17.58 

-12.04 

3B 

94.90 

6.91 

81.76 

15.71 

-13.54 

4A 

95.07 

8.39 

79.58 

15.30 

-15.19 

4B 

92.18 

13.60 

78.25 

24.26 

-13.59 

5A 

96.97 

5.92 

89.52 

13.04 

-6.87 

6A 

91.70 

11.98 

56.91 

26.89 

-33.28 

6B 

96.00 

6.43 

74.27 

18.76 

-21.29 

7A 

97.65 

5.50 

90.29 

10.03 

-6.93 

7B 

95.77 

7.72 

71.80 

15.85 

-22.50 

7C 

83.58 

18.23 

61.64 

21.72 

-20.40 

8A 

97.87 

3.86 

91.74 

10.16 

-5.69 

Forms 

60.37 

22.27 

61.89 

16.10 

2.90 

Manuals 

90.02 

14.87 

84.98 

16.61 

-5.43 

Total 

90.82 

4.64 

77.62 

8.48 

MM 

116.02 

10.39 

114.69 

11.29 

TIS 

60.10 

32.72 

71.55 

50.91 

N 

60 

89 

NOTE;  Site  adjustments  retain  performance  differences  due  to  aptitude  and  time 
in  service;  adjustments  are  added  to  site  A  hands-on  scores. 


Marine  Corps  manpower  experts  examined  the  hands-on  scores  and  site  differ¬ 
ences  to  determine  if  such  disparities  reflected  true  performance  differences  or  were 
merely  artifacts  of  the  hands-on  testing  process.  It  was  noted  that  for  two  tasks  that 
did  not  require  test  administrators  for  scoring  (forms  completion  and  use  of  tech¬ 
nical  manuals),  such  site  differences  were  not  foimd,  or  if  a  difference  did  occirr,  it 
was  typically  opposite  the  trend  noted  for  the  hands-on  tasks.  Job  knowledge  test 
scores  were  very  similar  for  the  two  sites,  and  the  sites  did  not  vary  in  terms  of  the 
mechanical  maintenance  (MM)  aptitude  composite.  Variations  in  time  in  service 
were  not  consistent  with  site  differences  in  hands-on  scores.  Inspection  of  the  step 
scores  revealed  that  particular  steps  were  scored  much  more  leniently  at  site  A  than 
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at  site  B.  For  these  reasons,  and  based  on  prior  Marine  Corps  experiences  and  their 
knowledge  of  the  hands-on  tests,  the  manpower  experts  concluded  that  score  differ¬ 
ences  overestimated  any  true  performance  differences  between  the  two  test  sites. 

Regression  analyses  were  conducted  for  each  task  within  each  MOS  to  estimate 
the  degree  to  which  task  performance  differences  were  due  to  aptitude  and  time-in- 
service  differences  between  the  two  sites.  Variance  in  performance  scores  due  to 
such  explainable  differences  should  be  retained  if  task  scores  are  adjusted  to  correct 
for  site  effects.  The  last  column  of  tables  9  through  11  provides  the  computed 
adjustments  for  each  task.  The  adjustments  are  apphed  to  the  task  scores  of  one 
site,  consistent  with  what  the  manpower  experts  believed  to  reflect  the  overall  level 
of  performance  in  the  fleet  for  these  MOSs. 

The  site  adjustments  preserved  the  relationships  among  primary  variables  of 
interest:  time-in-service,  aptitude,  and  hands-on  score.  For  some  tasks,  the  in¬ 
fluence  of  TIS  and  aptitude  on  task  performance  was  minimal,  so  the  adjustments 
essentially  reflect  the  differences  in  mean  task  performance.  Given  the  consistency 
of  differences  in  task  performance  by  site  and  across  MOS  and  the  consensus  among 
job  experts  that  the  differences  in  actual  proficiency  across  sites  were  minimal,  such 
adjustments  are  warranted.  The  site-adjusted  performance  scores  can  now  be 
aggregated  without  artificially  distorting  correlations  for  the  pooled  data. 

CONCLUSIONS 

The  procedures  that  were  followed  for  the  development  and  scoring  of  the 
mechanical  maintenance  hands-on  performance  tests  were  designed  to  ensure  that 
valid  inferences  could  be  drawn  firom  hands-on  test  scores  to  Marines’  job  perfor¬ 
mance.  ITSs  were  used  as  a  fovmdation  for  defining  job  requirements.  Marine  Corps 
subject  matter  experts  then  rated  each  job  requirement  for  its  representativeness  to 
the  job  as  a  whole.  Selection  among  those  tasks  that  were  judged  to  be  the  most 
representative  was  accomphshed  by  a  sampHng  procedure  that  maximized  the 
number  of  different  behavioral  elements  to  be  tested. 

After  tasks  were  selected,  tryout  testing  allowed  preliminary  assessments  of 
tasks’  feasibility,  time  limits,  and  procedures.  Retired  and  former  Marine  mechan¬ 
ics,  hired  to  serve  as  test  administrators,  were  given  two  weeks  of  training  to 
ensure  that  properly  rehable  and  valid  scoring  would  be  maintained.  During 
test  administration,  continual  monitoring  ensured  that  high  levels  of  inter-rater 
agreement  were  maintained,  and  that  scoring  criteria  did  not  drift  over  time. 
Hands-on  performance  data  were  entered  into  computers  at  each  test  site  to  identify 
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immediately  any  missing  data  and  to  allow  for  monitoring  of  test  administrator 
scoring  trends. 

Hands-on  performance  steps  were  scored  in  a  manner  that  ensxired  that 
experienced  mechanics  would  not  be  penalized  for  using  procedirres  that  were  more 
efScient  than  those  specified  by  the  technical  manuals.  After  testing  was  completed, 
data  were  transmitted  to  CNA  for  additional  quality  control  tests;  missing  data  were 
imputed  where  warranted.  Initial  analyses  examined  step-total  and  step-task  score 
correlations  to  identify  and  delete  steps  that  were  ambiguous  or  provided  poor 
measitrement. 

The  consistency  of  hands-on  measitrement  across  testing  sites  was  also 
examined.  The  relationship  between  tasks,  total  hands-on  score,  time  in  service, 
and  aptitude  were  found  to  be  very  consistent  for  the  two  sites  across  all  MOSs. 
However,  there  were  large  differences  in  the  percentage  of  correct  scores  for  the 
total  test  between  the  two  test  sites  for  all  helicopter  MOSs  (except  6113,  which  was 
tested  at  only  one  site).  Marine  Corps  manpower  experts  examined  hands-on  scores 
and  site  differences  to  determine  if  such  disparities  reflected  true  performance 
differences  or  were  merely  artifacts  of  the  hands-on  testing  process.  The  sites  did 
not  differ  on  job  knowledge,  or  on  the  two  tasks  that  did  not  require  test  admin¬ 
istrators  for  scoring;  nor  did  the  sites  vary  in  terms  of  mechanical  maintenance 
(MM)  composite.  Manpower  experts  concluded  that  score  differences  overestimated 
any  true  differences  in  performance  across  the  two  sites.  Therefore,  on  the 
recommendation  of  the  Marine  Corps  manpower  experts,  adjustments  were  made  to 
compensate  for  apparent  discrepancies  in  the  scoring  procedures  made  in  the  two 
testing  sites.  These  adjustments  prevented  artificial  distortions  for  the  pooled  data. 

The  goals  of  these  procedures  were  to  ensure  that  the  most  representative  tasks 
were  sampled  and  tested  in  an  objective  manner.  As  a  result  of  these  development 
and  scoring  procedures,  the  scores  that  come  from  the  mechanical  maintenance 
phase  of  the  Job  Performance  Measurement  (JPM)  Project  are  highly  reflective  of 
performance  in  the  actual  job  domain. 


1 


-25- 


REFERENCES 


[1]  CNA  Research  Contribution  570,  Developing  a  Competency  Scale  for 
Hands-on  Measures  of  Job  Proficiency,  by  Paul  W.  Mayberry,  Dec  1987 

[2]  American  Institute  for  Research  Report  AIR-70900-FR  02/91,  Develop  and 
Administer  Job  Performance  Measures  for  the  Mechanical  Maintenance 
Occupational  Areas,  Volume  I:  Test  Development,  by  J.  L.  Crafts  et  al., 
Feb  1991 

[3]  CNA  Research  Memorandum  91-120,  Data  Quality  for  the  Automotive 
Maintenance  Phase  of  the  Marine  Corps  Job  Performance  Measurement 
(JPM)  Project,  by  Neil  B.  Carey  and  Catherine  M.  Hiatt,  Aug  1991 

[4]  CNA  Research  Memorandum  91-221,  Data  Quality  for  the  Helicopter 
Maintenance  Phase  of  the  Marine  Corps  Job  Performance  Measurement 
(JPM)  Project,  by  Neil  B.  Carey  and  Catherine  M.  Hiatt,  Jan  1992 


-27- 


APPENDIX  A 


BEHAVIORIAL  ELEMENT  MATRICES  FOR  MARKER  CELLS  OF 
THE  AUTOMOTIVE  MECHANICS  JOB  REQUIREMENTS  DOMAIN 


Table  A-1.  Behavioral  element  matrix  for  “Inspect”  marker  cells  of  the  automotive  mechanic  job  requirements  domain 


Function: 


Not  tested 
Inspect 

Inspect  drive 


Tested 


Troubleshoot 


brakes 

train 

electrical  system 

Troubleshoot  engine 

Troubleshoot  hydraulic  system 

Inoperative 

Test  battery 

Engine  starts 

Excessive 

Winch 

hydraulic 

Inoperative 

Task 

No 

No 

Stoplights 

voltage  with 

but  fails  to 

oil  con¬ 

will  not 

low-oil-level 

folding 

tasks 

tasks 

inoperative 

STE/ICE 

keep  running 

sumption 

operate 

alarm  light 

boom 

(Vehicle): 

(813) 

(923) 

(923) 

(996) 

(LVS) 

(LVS) 

(LVS) 

Behavioral  elements 

Follow  specific  safety  instructions 

• 

X 

X 

X 

X 

X 

Look  for  wear  (loss  of  material,  deterioration)  • 

• 

X 

Look  for  damaged/defective  parts 
(cracks,  kinks,  breaks,  bends) 

Look  for  discoloration 

• 

X 

X 

X 

X 

X 

(material  is  Intact  but  off-color) 

Look  for  contamination  (material  is  intact) 

• 

X 

Check  for  proper  installation 

• 

X 

X 

X 

(mounting,  fasteners  tight) 

Look  for  missing  parts 

X 

X 

Look  for  fluid  leaks 

Listen  for  air  leaks 

X 

X 

X 

Locate  information  in  manual 

X 

X 

X 

X 

X 

X 

X 

Read  manual 

X 

X 

X 

X 

X 

X 

X 

Follow  general  safety  precautions 

X 

X 

X 

X 

X 

X 

X 

Fill  out  forms 

• 

X 

NOTE:  X  means  behavioral  element  was  tested. 

— >  means  behavioral  element  was  not  tested. 

•  means  behavioral  element  was  tested  in  context  of  another  task. 
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Table  A-2.  Behavioral  element  matrix  for  “Service”  marker  cells  of  the  automotive  mechanic  job  requirements  domain 


Function: 

Service  brakes 

Service  engine 

Bleed  system 

Task 

Master 

pressure 

Service 

Service 

cylinder* 

method* 

radiator 

oil  system 

(Vehicle): 

(813) 

(813) 

(LVS) 

(923) 

Behavioral  elements 

Follow  explicit  safety  instructions 

X 

X 

X 

Pour  fluid  into  reservoir 

X 

X 

Loosen  nut/bolt  with  wrench 

X 

X 

Pry  mechanism  to  loosenAighten 

X 

X 

Remove  piece  by  pulling  loose 

X 

X 

Install  piece  by  pushing 

X 

X 

Open/close  rotary  valve  by  hand 

X 

Tighten  nut/bolt  with  wrench 

X 

Wipe  surfaces  clean  of  dirt 

X 

X 

Lubricate  surface 

Open/close  valve  with  wrench 

X 

Prepare  bleeder  ball  for  use 

X 

Give  instructions  to  assistant 

X 

Follow  general  safety  precautions 

X 

X 

Select  proper  tools 

X 

X 

Fill  out  forms 

X 

X 

X 

X 

Locate  Information  in  manual 

X 

X 

X 

X 

Read  manual 

X 

X 

X 

X 

— > 


— > 
— > 


Allow  to  air  dry 

Wash/rinse  in  warm  (non-sudsy)  water 


(Continued  on  next  page) 


Table  A-2.  (Continued) 


Function: 

Service  brakes 

Service  engine 

Bleed  system 

Task 

Master 

pressure 

Service 

Service 

cylinder* 

method* 

radiator 

oil  system 

(Vehicle): 

(813) 

(813) 

(LVS) 

(923) 

Remove  dirt  and  rust  with  compressed  air 

X 

Loosen  drain  plug/gasket/cap 

X 

X 

Drain  fluid 

X 

X 

Install  gasket/drain/plug/cap 

X 

X 

Torque  bolt 

X 

Pour  fluid  into  reservoir  ■ 

X 

X 

Clean  item  with  dry-cleaning  solvent 

X 

X 

Dry  piece  with  compressed  air 

X 

X 

Test  alkalinity  of  fluid 

X 

Coat  seal  with  oil 

X 

Screw  on  oil  filter  by  hand 

\/ 

Follow  general  safety  precautions 

X 

X 

Select  proper  tools 

X 

X 

NOTE:  X  means  behavioral  element  was  tested. 

means  behavioral  element  was  not  tested. 

* .  "Service  Master  Cylinder"  and  "Bleed  Brake  System"  were  combined  for  testing. 
V  This  was  tested  in  LVS  radiator  task,  because  a  coolant  filter  was  screwed  on. 
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Table  A-3.  Behavioral  element  matrix  for  "Troubleshoot”  marker  cells  of  the  automotive  mechanic  job  requirements  domain 


Function: 

Troubleshoot  electrical  system 

Troubleshoot  engine 

Troubleshoot  hydraulic  system 

Inoperative 

Stoplight(s) 

Test  battery 

Excessive 

Engine  starts 

Winch 

hydraulic 

Inoperative 

Task 

are 

voltage  with 

oil 

but  fails  to 

will  not 

loW'Oil -level 

folding 

inoperative 

STE/ICE 

consumption 

keep  running 

operate 

alarm  light 

boom 

(Vehicle): 

(813) 

(923) 

(998) 

(923) 

(LVS) 

(LVS) 

(LVS) 

Behavioral  elements  common  to  both 
standard  and  STE/ICE  methods: 

Find  correct  starting  point  in  manual 

X 

X 

X 

X 

X 

X 

X 

Follow  steps  correctly 

X 

X 

X 

X 

X 

X 

X 

Make  correct  decisions 

X 

X 

X 

X 

X 

X 

X 

Operate  vehicle  controls  and  observe  results  X 

X 

X 

X 

Give  correct  instructions  to  assistant 

X 

Look  for  missing  parts 

□ 

X 

Look/feel  for  damaged/defective  parts 

□ 

X 

X 

Look/feel  for  loose  parts/improper 

D 

X 

X 

connections 

Look/feel  for  corrosion 

Look/feel  for  fluid  leaks 

D 

X 

X 

X 

Look/feel  for  wear* 

Check  for  contamination 

Test  specific  gravity  of  electrolyte 

Check  for  overheating 

Charge  batteries 

X 

• 

Standard  method 

Test  voltage  w/me ter— large  parts* 

Test  voltage  w/meter— small  parts  X  X 

(limited  connects) 

(Continued  on  next  page) 
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Table  A-3.  (Continued) 


Function:  Troubleshoot  electrical  system  Troubleshoot  engine  Troubleshoot  hydraulic  system 


Stoplight(s) 

Test  battery 

Excessive 

Engine  starts 

Winch 

Inoperative 

hydraulic 

Inoperative 

Task 

are 

voltage  with 

oil 

but  fails  to 

will  not 

low-oil -level 

folding 

inoperative 

STE/ICE 

consumption 

keep  running 

operate 

alarm  light 

boom 

(Vehicle): 

(813) 

(923) 

(998) 

(923) 

(LVS) 

(LVS) 

(LVS) 

Test  voltage  with  meter^small  parts  X 

(multiple  connects) 

Test  continuity  with  meter-small  parts  X  X 

(limited  connects) 

Test  continuity  with  meter-small  parts 
(multiple  connects) 


Test  resistance  with  meter 

Test  for  voltage  with  a  test  lamp  □ 

Test  circuits  by  opening  and  closing  them  □ 


STE/ICE  method 

Set  up  and  calibrate  STE/lCE  X 

Run  STE/ICE  test  X 

Look  for  corrosion 

Follow  explicit  safety  instructions 

Check  lines/assembly  for  blockage 

Test  fluid  levels 


X 


X 

X 

X 


X 

X 


X 

X 

X 


Check  indicator  for  color  reading  X 

Check  valve  position  X 

Check  for  air  In  system 
Pressurize  cooling  system 

Follow  general  safety  precautions  X  X  X  X  X 


(Continued  on  next  page) 


Table  A-3.  ((Continued) 


Function:  Troubleshoot  electrical  system  Troubleshoot  engine  Troubleshoot  hydraulic  system 


Inoperative 


Stoplight(s) 

Test  battery 

Excessive 

Engine  starts 

Winch 

hydraulic 

Inoperative 

Task 

are 

voltage  with 

oil 

but  fails  to 

will  not 

low-oil-level 

folding 

inoperative 

STE/ICE 

consumption 

keep  running 

operate 

alarm  light 

boom 

(Vehicle): 

(813) 

(923) 

(998) 

(923) 

(LVS) 

(LVS) 

(LVS) 

Connect/disconnect  connector 

Check  for  overloading  (excessive  weight) 

Check  for  binding  parts 

Install  jumper  wire 

Test  pressure  using  gauge 


X 

X 


Check  for  continuity 
Torque  nut 

— Remove  Insulation  from  wire 
^  Fill  out  forms* 

- 

NOTE:  X  means  behavioral  element  was  tested. 

— >  means  behavioral  element  was  not  tested. 

*  means  behavioral  element  was  tested  as  part  of  another  task  in  a  different  function. 
Q  not  in  manual  but  known  in  field. 


X  X 

X  X 
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Table  A-4.  Behavioral  element  matrix  for  “Test”  marker  cells  of  the  automotive  mechanic  job  requirements  domain 


Not  tested 

Tested 

Test 

Function:  electrical 

system 

Test 

hydraulic 

system 

Troubleshoot 
electrical  system 

Troubleshoot  engine 

Troubleshoot  hydraulic  system 

Inoperative 

Test  battery 

Engine  starts 

Excessive 

Winch 

hydraulic 

Inoperative 

Task 

No 

No 

Stoplights 

voltage  with 

but  fails  to 

oil  con¬ 

will  not 

low-oil-level 

folding 

tasks 

tasks 

inoperative 

STE/ICE 

keep  running 

sumption  ’ 

operate 

alarm  light 

boom 

(Vehicle): 

(813) 

(923) 

(923) 

(998) 

(LVS) 

(LVS) 

(LVS) 

Behavioral  elements 

Follow  specific  safety  instructions 

• 

• 

X 

X 

X 

X 

Set  switch  to  proper  position 

• 

X 

Interpret  instrument  reading 

• 

X 

X 

X 

Calibrate  meter  scale  (if  req.  by  meter) 

• 

X 

X 

Connect  probes  across  items 

• 

X 

Observe  operation  of  electrical  component 

• 

X 

Install  meter  probes  on  meter 

Instali/remove  jumper  wires 

• 

X 

X 

Operate  vehicle  controls  &  observe  results 

• 

X 

X 

X 

X 

Prepare  test  equipment 

• 

X 

Follow  general  safety  precautions 

X 

X 

X 

X 

X 

X 

Select  proper  tools 

X 

Fill  out  forms 

X 

Read  manual 

• 

X 

X 

Locate  information  in  manual 

• 

X 

X 

Place  lever  switch  in  proper  position 

• 

□ 

Operate  winch  manually 

Operate  winch  remotely 

• 

X 

Unplug  electrical  connector 

• 

X 

X 

(Continued  on  next  page) 
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Table  A-4.  (Continued) 


Not  tested 

Tested 

Test 

Test 

Function: 

electrical 

hydraulic 

Troubleshoot 

system 

system 

electrical  system 

Troubleshoot  engine 

Troubleshoot  hydraulic  system 

Inoperative 

Test  battery 

Engine  starts 

Excessive 

Winch 

hydraulic 

Inoperative 

Task 

No 

No 

Stoplights 

voltage  with 

but  fails  to 

oil  con¬ 

will  not 

low-oil-level 

folding 

tasks 

tasks 

inoperative 

STE/ICE 

keep  running 

sumption 

operate 

alarm  light 

boom 

(Vehicle): 

(813) 

(923) 

(923) 

(998) 

(LVS) 

(LVS) 

(LVS) 

Remove  pressed-on  component 

« 

□ 

Set  multimeter  rotary  switch  to  proper  position 

• 

X 

X 

Connect  multimeter  leads  to  proper  contacts 

• 

X 

X 

Read  multimeter  correctly 

Interpret  dipstick  fluid  level  reading 

• 

X 

X 

X 

Connect  jumper  wire 

• 

X 

X 

NOTE:  X  means  behavioral  element  was  tested. 

— >  means  behavioral  element  was  not  tested. 

•  means  behavioral  element  was  tested  in  context  of  another  task. 

□  not  in  manual  but  known  in  field. 
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Table  A-5.  Behavioral  element  matrix  for  “Adjust/Align”  marker  cells  of  the  automotive  mechanic  job  requirements  domain 


Function: 

Adjust/align  steering  system 

Power-steering 

Task 

Power-steering 

assist  cylinder 

pump  belt 

(travel  adjustment) 

Toe  in/out 

(Vehicle): 

(1008) 

(813) 

(998) 

Behavioral  elements 

Loosen  bolt/screw  (do  not  remove) 

X 

X 

X 

Loosen  nut  (do  not  remove) 

X 

X 

Tighten  belt  with  pry  bar 

Tighten  screw/bolt 

X 

X 

Tighten  nut 

X 

X 

Read  belt  tension  with  gauge 

X 

Install  adjusting  clamp 

X 

Rotate  assembly  clockwise 

X 

X 

Rotate  assembly  counterclockwise 

X 

X 

Position/remove  gauge  between  wheels** 

X 

Move  scale  (gauge)/(string)/(chalk  line) 

X 

Read  scale/measurement/(string)/(chalk  line) 

X. 

X 

Loosen  clamp 

X 

X 

Loosen  wheel  with  puller  tool 

X 

Realign  steering  wheel 

X 

Torque  nut/bolt/screw 

X 

X 

Remove  and  install  cotter  pin* 

Jack  wheels  up/down*  • 

Give  instructions  to  assistant 

X 

Follow  general  safety  precautions 

X 

X 

X 

(Continued  on  next  page) 


Table  A-5.  (Continued) 


Function: 

Adjust/align  steering  system 

Power-steering 

Task 

Power-steering 

assist  cylinder 

pump  belt 

(travel  adjustment) 

Toe  in/out 

(Vehicle): 

(1008) 

(813) 

(998) 

Select  proper  tools 

X 

X 

X 

Fill  out  forms 

X 

X 

X 

Locate  information  in  manual 

X 

X 

X 

Read  manual 

X 

X 

X 

NOTE:  X  means  behavioral  element  was  tested. 

means  behavioral  element  was  not  tested. 

*  means  behaviorial  element  was  tested  as  part  of  another  task  in  a  different  function. 


Table  A-6,  Behavioral  element  matrix  for  "Repair  and  Replace”  marker  cells  of  the  automotive  mechanic  job  requirements  domain 


Function: 

Repair  and  replace 

brake  system 

Repair  and  replace  drive  tram 

Task 

Parking 

Neutral  start 

Rear  propeller 

Universal 

Runflat 

Brake  shoes 

brake  cable* 

switch 

shafts* 

joints 

assembly 

(Vehicle): 

(1008) 

(998) 

(923) 

(998) 

(998) 

(998) 

Behavioral  elements 

Follow  explicit  safety  instructions 

X 

X 

X 

X 

Withdraw/pull  forward  on  mechanism 

X 

X 

X 

X 

X 

X 

Unhook/remove  expansion  spring 

X 

Remove  coil  compression  spring 

X 

Depress  tabs  on  nut 

X 

Remove  washer/bushing 

X 

X 

X 

Loosen  nut  by  hand 

X 

X 

X 

X 

X 

Loosen  nut  by  wrench 

X 

X 

X 

X 

Depress/push  mechanism  by  hand 

X 

X 

X 

X 

Tighten  nut  by  wrench 

X 

X 

X- 

X 

Tighten  nut  by  hand 

X 

X 

X 

X 

X 

Install  washer/bushing 

X 

X 

X 

X 

Position  tabs  on  nut 

X 

Install  coil  compression  spring 

X 

X 

Rehook/install  expansion  spring 

X 

Disconnect  fluid  line  using  line  wrench 
Connect  fluid  line  using  line  wrench 
Pry  mechanism  to  either  loosenAighten 
Torque  nut 
Loosen/tighten  screw 


X 


X 

X 


X 

X 


X 

X 

X 


(Continued  on  next  page) 
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Table  A-6.  (Continued) 


Function: 

Repair  and  replace 
brake  system 

Repair  and  replace  drive  train 

Task 

Parking 

Neutral  start 

Rear  propeller 

Universal 

Runflat 

Brake  shoes  brake  cable* 

switch 

shafts* 

joints 

assembly 

(Vehicle): 

(1008)  (998) 

(923) 

(998) 

(998) 

(998) 

Compress  mechanism  with  C-clamp 

Follow  general  safety  precautions 

X  X 

X 

X 

X 

X 

Select  proper  tools 

X  X 

X 

X 

X 

X 

Fill  out  forms 

X  X 

X 

X 

X 

X 

Locate  information  in  manual 

X  X 

X 

X 

X 

X 

Read  manual 

X  X 

X 

X 

X 

X 

R  &  R  cotter  pin 

Remove  gaskets/sealsAD  rings 

X 

X 

X 

X 

Install  gaskets/seals/0  rings 

X 

X 

X 

Clean  parts 

X 

X 

X 

Operate  vehicle  controls 

X 

X 

Remove  and  install  retaining  clips 

X 

X 

Apply  grease  or  sealant 

Connect/disconnect  male/female  connector 
Install  grease  packets 

X 

X 

X 

X 

X  means  behavioral  element  was  tested. 

— >  means  behavioral  element  was  not  tested. 

*  means  "Parking  brake  cable"  and  "Rear  propeller  shafts"  were  combined  for  testing. 

APPENDIX  B 


SKILL-BY-BEHAVIORIAL-ELEMENT  MATRIX 
FOR  HELICOPTER  MECHANICS 


APPENDIX  B 


SKILL-BY-BEHAVIORIAL-ELEMENT  MATRIX 
FOR  HEUCOPTER  MECHANICS 


Lube 

systems 

Main 

gearbox 

Accessory 

gearbox 

Drive  shaft 
system 

Manipulate  electrical  switches 

X 

X 

X 

Disconnect  plugs/switches 
Disconnect  wires 

X 

X 

X 

X 

X 

Check  advisory  light  (VI) 

X 

X 

X 

Reconnect  plugs/switches 

X 

X 

X 

Connect  wires 

X 

X 

X 

Read  gauges 

X 

X 

X 

Check  fluid  levels  (VI) 

X 

X 

X 

Check  for  leads  (VI) 

X 

X 

X 

Check  for  dirt  (filters)  (VI) 

X 

X 

X 

Remove  accessories 

X 

X 

X 

X 

Connect  accessories 

X 

X 

X 

X 

Check  fluids  for  contamination 

X 

X 

X 

Perform  oil  analysis 

X 

X 

X 

Drain  fluids 

X 

X 

X 

SOURCE:  American  Institutes  for  Research  Report  AIR-70900- FR  02/91 .  Develop  and 

Administer  Job  Performance  Measures  for  the  Mechanical  Maintenance  Occupational 

Area,  Volume  1:  Test  Development,  J.  L  Crafts  etal..  Feb  1991. 
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APPENDIX  C 


CONTENT  TESTED  FOR  EACH  MARKER  CELL 
FOR  HELICOPTER  MECHANICS,  BY  MOS 


APPENDIX  C 


CONTENT  TESTED  FOR  EACH  MARKER  CELL 
FOR  HELICOPTER  MECHANICS,  BY  MOS 


i  '  ''  ' '' 

i-x-x  •  * 

mi. 

0$ 

1  '  1 

Behavioral  element 

6112 

6113 

6114 

6115 

Measure/record  distance  using  ruler 

• 

• 

Connect  components 

• 

• 

Connect  accessories 

• 

• 

Inspect  for  POD  and  cleanliness 

• 

• 

Apply  electrical  power 

• 

• 

Remove  electrical  power 

• 

• 

Record  information  on  maintenance  forms 

m 

• 

Use  precision  measuring  equipment 

• 

• 

Use  special  tools 

• 

• 

Clean  with  cloth 

• 

• 

Safety  wire 

m 

• 

• 

Turn  on  power  using  APP 

• 

m 

Turn  off  power  [turn  off  APP] 

• 

m 

Disconnect  accessories 

• 

• 

• 

Adjust  component 

• 

• 

• 

Lift  blade 

m 

Lower  blade 

m 

Check  for  proper  engagement 

% 

• 

• 

Remove  gaskets/seals/O-rIngs 

• 

• 

• 

Disconnect  hosesAubes/lines 

• 

• 

• 

Loosen  check  nuts 

• 

Install  gasket/seals/O-rings 

• 

• 

• 

Connect  hoses/tubes/lines 

• 

• 

• 

Adjust  accessories 

• 

• 

• 

Straighten  tangs  on  key  washer 

Bend  tangs  on  key  washer 

Check  switches/valves/levers  for  proper  position 

• 

• 

• 

• 

Cap  off  lines 

• 

• 

• 

• 

Remove/inspect  chip  detectors 

• 

• 

• 

• 

Check  for  thread  engagement 

Adjust  rods/bearings 

• 

• 

• 

• 

Use  antifreeze  compound 

Loosen  nuts 

• 

• 

• 

• 

Tighten  lock-nut  by  hand 

• 

• 

{Continued  on  next  pege) 
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'  V  ■ 

Behavioral  elements 

6112 

6113 

6114 

6115 

Manipulate  electrical  switches 

• 

• 

• 

• 

Disconnect  plugs/switches 

• 

• 

• 

• 

Manipulate  flight  controls 

• 

• 

Check  advisory  light 

• 

• 

9 

Reconnect  plug 

• 

• 

• 

• 

Read  gauges 

• 

• 

Adjust  friction 

• 

Check  that  accessories/components  are  within  tolerance 

• 

• 

• 

m 

Use  spring  scale  to  check  pull 

• 

Check  for  play 

• 

• 

• 

• 

Check  for  fluid  leaks 

• 

• 

• 

m 

Visually  inspect  components  for  integrity/damage 

• 

• 

• 

m 

Check  mechanical  operation 

m 

• 

• 

• 

Check  for  corrosion 

m 

• 

m 

• 

Select  maintenance  tools 

• 

• 

• 

m 

Manipulate  maintenance  tools 

• 

• 

• 

• 

Check  for  binding 

Select  correct  component  to  troubleshoot 

• 

• 

m 

• 

Use  dial  indicators 

• 

• 

• 

• 

Use  manuals 

• 

• 

• 

• 

Shims 

m 

m 

m 

• 

Rigs 

• 

Torque 

• 

• 

• 

• 

Disconnect  control  rods/bellcranks/supports 

• 

• 

m 

m 

Remove  cotter  pins/nuts/washers/bushings/clamps/shims 

• 

• 

• 

• 

Remove  large  bolts 

• 

• 

• 

• 

Install  large  bolts 

• 

• 

• 

• 

Remove  component 

• 

• 

• 

• 

Remove  accessories 

• 

• 

• 

• 

Align  components  for  installatton/removal 

• 

• 

• 

• 

Install  cotter  pins/nuts/washers/bushings/clamps/shims 

• 

• 

• 

• 

Align  accessories  for  instailation/removal 

• 

• 

• 

• 

Connect  control  rods/bellcranks/supports 

• 

• 

• 

• 

Apply  primer  (spray) 

Use  gauge  to  check  gaps  or  clearance 

• 

• 

• 

• 

SOURCE:  American  Institutes  for  Research  Report  AIR-70900-FR  02/91.  Develop  and  Administer  Job 

Performance  Measures  for  the  Mechanical  Maintenance  Occupational  Area,  Volume  1 :  Test  Develofy 

menu  J.  L  Crafts  et  ai.,  Feb  1991 . 
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